HADOOP-19658. ABFS:Create and rename idempotency for FNS Blob#7914
Merged
anujmodi2021 merged 8 commits intoapache:trunkfrom Sep 3, 2025
Merged
HADOOP-19658. ABFS:Create and rename idempotency for FNS Blob#7914anujmodi2021 merged 8 commits intoapache:trunkfrom
anujmodi2021 merged 8 commits intoapache:trunkfrom
Conversation
anmolanmol1234
changed the title
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
| AbfsRestOperation statusOp = null; | ||
| try { | ||
| // Check if the file already exists by calling GetPathStatus | ||
| statusOp = getPathStatus(path, false, tracingContext, null); |
Contributor
There was a problem hiding this comment.
In case of override true, flow might come here with already a Head call done on path.
Can we avoid this head call in that case?
| assumeHnsDisabled(); | ||
| // Create a spy of AzureBlobFileSystemStore | ||
| AzureBlobFileSystemStore store = Mockito.spy(fs.getAbfsStore()); | ||
| assumeBlobServiceType(); |
Contributor
There was a problem hiding this comment.
We can move all assume before any other statement
| // Create a spy of AzureBlobFileSystem | ||
| try (AzureBlobFileSystem fs = Mockito.spy( | ||
| (AzureBlobFileSystem) FileSystem.newInstance(getRawConfiguration()))) { | ||
| assumeHnsDisabled(); |
Contributor
There was a problem hiding this comment.
Same here, move all assume to first few lines
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
anujmodi2021
approved these changes
Sep 1, 2025
Contributor
anujmodi2021
left a comment
anujmodi2021
left a comment
There was a problem hiding this comment.
+1
LGTM
Some nits added
| /**Flag to enable/disable sending client transactional ID during create/rename operations: {@value}*/ | ||
| public static final String FS_AZURE_ENABLE_CLIENT_TRANSACTION_ID = "fs.azure.enable.client.transaction.id"; | ||
| /**Flag to enable/disable create idempotency during create operation: {@value}*/ | ||
| public static final String FS_AZURE_ENABLE_CREATE_IDEMPOTENCY = "fs.azure.enable.create.idempotency"; |
Contributor
There was a problem hiding this comment.
This is only for Blob Idempotency, may be we can keep config name accordingly
|
|
||
| public static final boolean DEFAULT_FS_AZURE_ENABLE_CLIENT_TRANSACTION_ID = true; | ||
|
|
||
| public static final boolean DEFAULT_FS_AZURE_ENABLE_ENABLE_CREATE_IDEMPOTENCY = true; |
Contributor
There was a problem hiding this comment.
Typo, ENABLE added twice
| key = "HTTP Response"; | ||
| } | ||
| String values = StringUtils.join(";", entry.getValue()); | ||
| List<String> valuesList = entry.getValue(); |
Contributor
Author
There was a problem hiding this comment.
Due to null pointer exceptions on enabling AbfsIoUtils logging if value is null.
|
🎊 +1 overall
This message was automatically generated. |
| public boolean isConditionalCreateOverwriteEnabled() { | ||
| // If either the configured FS service type or the ingress service type is BLOB, | ||
| // conditional create-overwrite is not used. | ||
| if (getFsConfiguredServiceType() == AbfsServiceType.BLOB |
Contributor
There was a problem hiding this comment.
why is this change needed?
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
manika137
approved these changes
Sep 2, 2025
Contributor
Author
============================================================
|
anmolanmol1234
added a commit
to anmolanmol1234/hadoop
that referenced
this pull request
Sep 3, 2025
…#7914) Contributed by Anmol Asrani
anujmodi2021
pushed a commit
that referenced
this pull request
Oct 16, 2025
) (#7929) Contributed by Anmol Asrani
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change introduces idempotency for create operation in the ABFS driver when using FNS-Blob. The update strengthens the client’s retry semantics to ensure that transient backend failures (e.g., HTTP 500) or ambiguous outcomes (where the client receives an error, but the operation has already succeeded on the service) are handled safely. This PR also includes test cases to validate the idempotency of rename operations.